iProbe: A Lightweight User-Level Dynamic Instrumentation Framework

Application tracing in production systems requires dynamic and flexible instrumentation mechanisms with lowoverhead. Tracing tools may be required to be started at anytime, and it can take potentially long time periods to collect enough information, but at the same time should not adversely affect service quality. Existing user-space code monitoring solutions are either inflexible developer-driven static instrumentation which require manual effort, or black-box dynamic instrumentation techniques which are flexible but have high overhead. To solve this problem, we introduce a new hybrid instrumentation technique for user-space code monitoring called iProbe, which is flexible and has low overhead. iProbe takes a novel 2-stage design, and offloads much of the dynamic instrumentation complexity to an offline compilation stage. It leverages standard compiler flags to introduce “place-holders” for hooks in the program executables. Then it utilizes an efficient userspace HotPatching mechanism which dynamically instruments the functions to be traced and enables execution of instrumented code in a safe and secure manner. We implemented iProbe as a dynamic application profiling framework. In its evaluation on micro-benchmarks and SPEC CPU2006 benchmark applications, the iProbe prototype achieved the instrumentation overhead an order of magnitude lower than existing state-of-the-art dynamic instrumentation tools like SystemTap and DynInst. We also built a hardware event profiling tool based on iProbe, and were able to obtain function-level hardware event breakdown on SPEC CPU2006 applications with controlled performance overhead (e.g., under 5%).